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Introduction 

KIPP Ujima Village Academy is a member of the KIPP 
(Knowledge is Power) National Network of schools. The 
Baltimore KIPP serves approximately 300 students in 
Northwest Baltimore in grades 5-8; the school opened in 
Fall 2002 and converted to a charter school in 2005. 



The Baltimore KIPP Ujima Village Academy, 2002-2006: 

A Longitudinal Analysis of Student Outcomes 

Executive Summary 

This study analyzes four cohorts of 5* grade students in the Baltimore City Public School 
System (BCPSS) from 2002-03, 2003-04, 2004-05, and 2005-06. The primary group of interest 
is 5* grade students at the BCPSS KIPP school for each of these cohort years. We then use data 
available from BCPSS to describe and trace the trajectory of these KIPP students over time, 
compared to 5^’’ grade cohort groups from the feeder KIPP schools in each cohort year (defined 
as those 4* graders the prior year who were promoted to 5* grade as each KIPP cohort began 5* 
grade). Major findings were: 

□ KIPP students were less likely to have special education status than their feeder school 
comparison group. Otherwise, they were demographically similar (except for two 
cohorts in which KIPP had a higher proportion of females). 

□ The first and fourth cohort of KIPP students (5* graders in 2002-03 and 2005-06) did not 
differ significantly from their feeder school comparison group in prior 4* grade 
achievement. KIPP students had significantly higher mathematics scores than the 
comparison group in Cohorts 2 and 3, and also significantly higher reading scores than 
the comparison group in Cohort 2. (The impact of students who entered KIPP from 
outside BCPSS and thus had no prior achievement scores cannot be measured in these 
analyses.) 

□ Even when pre-existing differences between KIPP and comparison students are 
controlled in statistical analyses, KIPP students generally outperformed comparison 
school students on achievement measures. In particular: 

o KIPP students significantly outperformed their feeder school comparison group in 
5th grade mathematics every year (even controlling for higher prior achievement 
in math). But KIPP students did not outperform their feeder school comparison 
group in 5* grade reading achievement. 

o In grades 6 to 8, when most of the comparison group students attended large 
middle schools, KIPP students outperformed comparison students in reading and 
in math. In most cases, the KIPP effect was significantly positive even when 
students who had transferred out of KIPP and were still at the on-time grade level 
were included in the analyses as “ever KIPP” students. The necessary exclusion 
of retained students from test score analyses may have an unmeasured impact on 
these reported results. 

□ Attrition from the KIPP program was not trivial, and students who left KIPP had lower 
test scores than those who stayed at KIPP. Attrition was also generally higher among 
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males than females. KIPP students had significantly lower on-time promotion rates in 

Cohorts 2 and 3 than comparison group students, but there was no difference in Cohort 1 . 

Though KIPP and comparison students were similar in 4* grade achievement and 
attendance, as well as on demographic variables, it is important to note that it was not possible to 
measure other important differences between KIPP and comparison students. In particular, 
differences in family support variables (parental education, amount of parental interaction with 
students on academic and other activities, etc.) could not be measured, and could contribute 
significantly to the later achievement differences between the groups. 

It is likely that several components of the KIPP program contributed to higher student 
achievement: the longer school day and other additional hours of instruction, high quality 
curriculum and instruction (especially in mathematics), and positive school climate (facilitated 
by smaller numbers of students than in comparison students’ schools, fewer behavioral problems, 
etc.). Scaling up these components throughout the district is an attractive proposal, but would 
likely be very costly. It is also important not to ignore the attrition from the KIPP program, as 
well as the relatively higher retention in grade rates for KIPP students. Assuring that a KIPP 
model could succeed on a large scale and sustain such high rates of achievement would be an 
extremely challenging task. Qualitative studies that probe into reasons for the attrition among 
KIPP students would be a useful research contribution as district policymakers make decisions 
on expanding the number of KIPP schools. 

It is possible that keeping middle grades students in smaller learning environments 
(newly converted K-8 schools rather than middle schools) will address the school climate issue, 
but the primary challenge is to find ways to ensure high quality instruction in every classroom, 
every day. In addition, more ways of providing extra help for struggling students need to be 
incorporated into district- and school-level planning of instructional delivery. The KIPP Ujima 
Village Academy in Baltimore has shown that high quality instruction and extra learning time in 
a positive school environment does make a difference in student achievement. The challenge is 
to find ways to make this a reality for most urban students, rather than for just a few. 
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The Baltimore KIPP Ujima Village Academy, 2002-2006: 

A Longitudinal Analysis of Student Outcomes 

Background 

Low academic performance by students in urban middle schools has led to a variety of 
reform efforts, including the creation of new K-8 schools (returning to an earlier model of 
schooling in the U.S.) as well as charter-type schools serving middle grades students. The 
Baltimore City Public School System (BCPSS), like other large city school districts, has 
explored both of these options recently. Under the district’s “New Schools Initiative,” several 
new schools serving middle grades students have been created. A recent Abell Foundation 
report (2006) notes in particular the high levels of academic achievement of students at the KIPP 
(Knowledge is Power Program) Ujima Village Academy, compared to students at city middle 
schools and K-8 schools. 

Created in 1994 by two former Teach for America teachers, the Knowledge Is Power 
Program (KIPP) began with a fifth grade program in Houston, expanded next to New York City, 
and as of spring 2007 included 52 schools in 16 states and the District of Columbia serving 
12,000 students. KIPP schools have primarily enrolled underserved urban minority students 
(63% African American, 33% Hispanic) in grades 5 to 8, though the program has now expanded 
into rural areas and both elementary and high school gradespans. While principals at each KIPP 
school are free to select or design their own curriculum, a standard feature across all KIPP 
schools is the increased learning time (extended day, alternate Saturdays, and three weeks of 
summer school), characterized by KIPP leaders as “60% more than average public school 
students” (KIPP, 2007). 

Besides numerous journalistic articles praising the KIPP program (e.g., Choi, 2003; 

Izumi, 2004), several recent reports have analyzed achievement outcomes for students at KIPP 
schools. A KIPP-commissioned study conducted by the Educational Policy Institute (2005) 
found large NCE gains on the Stanford Achievement Test (SAT 9/10) for 5th graders at 24 KIPP 
schools in mathematics, as well as somewhat smaller gains in language and reading. In addition, 
several independent evaluation reports have noted positive achievement effects for KIPP schools. 
Only one study thus far uses longitudinal student level data for both KIPP and comparison group 
students (Gallagher & Rossi, 2005), finding stronger effects for math than reading and English 
language arts in 5th grade and nearly equal effect sizes for all subjects in 6th grade. This 
longitudinal study of Memphis students noted that while both KIPP and comparison lost NCE 
points between 5th and 6th grade (about equally) in reading and math, KIPP students still 
outperformed the matched comparison group. Other studies of KIPP in Houston, North 
Caroline, and District of Columbia (Doran & Drury, 2002), San Erancisco (David et ah, 2005), 
and Denver (Anderson & DeCesare, 2006) also found positive achievement effects of the KIPP 
program, though their authors noted weaknesses in their methodologies (lack of control groups 
or ability to assess achievement growth over time) that limited the conclusions they were able to 
draw. 
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The need to investigate attrition and in-grade retention patterns is an issue that KIPP 
acknowledges (KIPP, 2007). As David et al. (2006, p. 63) note: “In-grade retention and attrition 
data from two KIPP schools indicate that their student cohorts change significantly from year to 
year,” and these evaluators highlight the need to determine whether there are systematic patterns 
in attrition (particularly loss of lower-achieving and/or poorly behaved students). While 
observers have made claims in internet blog (San Francisco Schools Blog, 2007) regarding 
attrition at KIPP schools, this issue requires more systematic study. 

The following research study examines four cohorts of KIPP students in Baltimore, 
together with a comparison groups from the same feeder elementary schools as each KIPP 
cohort. In particular, this research study addresses whether there were pre-existing differences 
between KIPP students and other BCPSS students that could help to explain the higher academic 
achievement levels of the KIPP school compared to other schools in aggregate level analyses. In 
addition, this research study focuses on issues such as attrition (and new student replacement) at 
the KIPP school. We systematically compare academic achievement outcomes of KIPP students 
with students at the same grade level at other types of schools. 

Study Research Questions and Methodology 

We identify four cohorts of 5* grade students in the Baltimore City Public School System 
(BCPSS) from 2002-03, 2003-04, 2004-05, and 2005-06. The primary group of interest is 5* 
grade students at the BCPSS KIPP Ujima Village Academy for each of these cohort years. (The 
school began with a cohort of 5* graders in 2002-03 and added a grade a year through 8^** grade 
in 2005-06.) We then use data available from BCPSS to describe and trace the trajectory of 
these KIPP students over time, compared to 5^’’ grade cohort groups from the feeder KIPP 
elementary schools in each cohort year (defined as those 4* graders the prior year who were 
promoted to 5* grade as each KIPP cohort began 5* grade). We show the group of cohorts and 
their progression over time schematically in Figure 1. 



Figure 1. Cohort Progression Over Time 





2002-2003 

Grade 


2003-2004 

Grade 


2004-2005 

Grade 


2005-2006 

Grade 


Cohort 1 


5 


6 


7 


8 


Cohort 2 




5 


6 


7 


Cohort 3 






5 


6 


Cohort 4 
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In particular, we define each 5* grade cohort as those students enrolled in KIPP on 
September 30* of the 5* grade year (excluding those who transfer out before September 30, 
including those who transfer in by September 30). We then divide this group each year into 
those who stayed all year, and those who transferred out of KIPP before the end of the year. In 
addition, we construct a group of students who transferred into KIPP (5* graders after September 
30 and 6* graders) for comparative analysis. We then compare characteristics (demographic, 
prior school achievement and attendance) and outcomes (promotion to next grade, attendance, 
test scores) for these groups of students using variable-appropriate statistical tests to determine 
whether differences between the groups are statistically significant. 
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All analyses with Maryland School Assessment (MSA) test score data are conducted only 
with students at the same grade level and for those students promoted on the regular schedule 
(necessarily excluding retained students from test score data analyses). In addition, all 
comparisons using MSA scale score data are made only for the same testing year; no cross-year 
comparisons can be made due to the design and construction of the MSA tests (CTB/McGraw- 
Hill, 2006). 

Characteristics of KIPP Students 

Previous School 

Eligibility for enrollment in Baltimore’s KIPP Ujima Village Academy is linked to 
having an address zoned to particular feeder elementary schools in Northwest Baltimore.^ Figure 
2 summarizes the distribution of 4* grade feeder schools for each of the grade cohorts. While 
the large majority of students each year came from the designated feeder elementary schools in 
the geographic area that KIPP serves, each year there were also students in the original 5* grade 
cohort whose last 4* grade school was not one of the feeder elementary schools (from which the 
comparison group for this study was composed). One likely explanation is that these students 
had an address change that then made them eligible to enroll in KIPP. In the 2002-03 cohort of 
5* graders, all 4 students who were not from the feeder elementary schools were from outside 
BCPSS (did not attend a BCPSS school in 2001-02). Each of the subsequent 5* grade cohorts 
included students from other (non-feeder) BCPSS schools as well outside the district. The 
percentage of non-feeder students ranged from 5% to 15%, depending on the year. Another 
notable pattern in these data is the higher concentration in later cohorts of students from schools 
with lower poverty rates (see Appendix A). In particular. Cross Country Elementary, a non-Title 
I school, sent more students for KIPP's 2004-05 5th grade cohort than any other feeder 
elementary did. 



* See Appendix A for a summary of how the list of elementary feeder schools changed over time. 

^ Two students in Cohort 2 and one in Cohort 4 had previously attended a KIPP feeder school even though their last 
school before KIPP was a non-feeder. 



5 




0 ) 




o 

CO 



o 

O 

o 

^ o 
^ o 



0 

> 

O 



•o 

■Q 

(/) "> 



o 

o 

u 

0) 



■o 

0) 

0) 



O 2 

■a 1 

0 {" 

O - 

U- s 

O 0 

C " 

0) 



0 ) 

Q ^ 

csi 

Q) 

i- 

3 

G) 



O 




LO 

CM 



Q. 

Q. 



O 

c 



c 

d) 



c 

d) 

"O 

3 

(0 

o 

d) 

E 

3 



CO 

CO 








CL 


CM 




1— 


O 


CO 


d 


CT> 


CD 




CO 


00 


O 


O 


"o 


= 


3 


■o 


o 


r 


O 


to 

O 


o 

CO 


CO 

3 




§ 


c 

o 


O 

aS 



§ 



































OO 


CO 


1^ 


CO 


1— 


U) 


0 


0 


00 


d 




CM 




1< 






D 


00 


in 


a) 


Of) 


00 


CO 


r-- 


















c 




w 


to 


-p 




c 


_0) 




CO 








CO 


3 


•— 


D) 

_C 


CL 


U) 

'o 


-n 


'3 


CL 

CD 


0 

E 


c 

0) 








c 


0 


> 


c 


C3 


CO 


CO 


■O 


>, 


0 


CD 

0 


CO 


§ 


O 


o 






0 


0 






(1) 












o 

































X O 
>, ^ 
t c 
0) 0 
^ CD 
O 



CM 

cj) d 
CO CO CO 



o 

O 



o 

6 



CO CO 



§ ^ 
D) i 
.E CL 



















0 


0 


CO 


LO 


c\i 


CO 


tri 


CO 


0-) 


0-) 


00 


cn 


















c/) 


c 


(1) 


Cl) 


CD 


3 


0 


c 


C3) 


r 


0 


_l 


X 


CO 


(D 

O 


E 


o 




b 




to 

O) 

c 

CO 

_l 


5 

_C0 

3 

LU 


0 

0 

CD 

O) 

■D 

LU 





(90-t7003 '^oun\ psonpaj/aaj^ jo^ ^iqiBna siuapms p %) looqas 



6 



Demographic Characteristics 



Table 1 summarizes demographic characteristics for the original KIPP students in each 
cohort with comparison students from the designated feeder elementary schools^ who were also 
promoted to 5* grade. As the table illustrates, KIPP students did not differ notably from 
comparison students in race or free lunch status. KIPP had a higher proportion of female 
students than the comparison group in two of the cohorts, but a lower proportion in the last 
cohort. The KIPP cohorts differed most from comparison cohorts in the lower proportion of 
special education students (though this number grew over time in the original cohort groups). 

Prior Academic Characteristics 

As Table 1 indicates, KIPP students had only slightly higher prior (4* grade) attendance 
rates than comparison students. The first and fourth cohorts of KIPP students (5* graders in 
2002-03 and 2005-06) did not differ significantly from their feeder school comparison group in 
prior 4^’’ grade achievement. KIPP students had significantly higher 4* grade mathematics scores 
than the comparison group in Cohorts 2 and 3, and also significantly higher 4* grade reading 
scores than the comparison group in Cohort 2 (see Table 2). The impact of students who entered 
KIPP from outside BCPSS and thus had no prior achievement scores cannot be measured in 
these analyses. 



Table 1. Summary of Demographic Characteristics and Prior Attendance for KIPP and 

Comparison Students 

Cohort 1 Cohort 2 Cohort 3 Cohort 4 

S*** grade 2002-03 5“* grade 2003-04 grade 2004-05 grade 2005-06 



Original Comparison Original Comparison Original Comparison Original Comparison 
KIPP Group KIPP Group KIPP Group KIPP Group 
Cohort Cohort Cohort Cohort 



% Female 


49.4% 


48.0% 


59.6% 


50.8% 


54.1% 


50.6% 


46.6% 


50.1% 


% Special Ed 


5.3% 


15.0% 


5.7% 


16.1% 


8.8% 


14.9% 


12.5% 


17.6% 


%FRL 


89.9% 


84.0% 


93.0% 


90.2% 


82.4% 


86.1% 


89.8% 


87.9% 


% African- 
American 


100% 


99.4% 


100% 


99.3% 


100% 


97.9% 


97.7% 


98.5% 


4* grade 
attendance 


96.0% 


94.4% 


94.9% 


93.6% 


96.8% 


94.1% 


96.4% 


95.5% 



Note: Statistically significant differences (p <.05) indicated in bold font. 



^ See Appendix A for information on which feeder school students were included in the comparison group for each 
cohort. 
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Table 2. Summary of Prior (4* grade) Achievement Scores for KIPP and Comparison 

Students 

Cohort 1 Cohort 2 Cohort 3 Cohort 4 

5* grade 2002-03 5“* grade 2003-04 5“* grade 2004-05 5“* grade 2005-06 

Original Comparison Original Comparison Original Comparison Original Comparison 





KIPP 

Cohort 


Group 


KIPP 

Cohort 


Group 


KIPP 

Cohort 


Group 


KIPP 

Cohort 


Group 


Reading 


45.7 


44.2 


49.7 


42.6 


385.8 


380.7 


388.8 


379.4 


Math 


47.9 


46.3 


52.3 


45.6 


385.1 


370.0 


387.0 


377.2 



Note: Statistically significant differences (p <.05) indicated in bold font. 

Note: Figures for Cohorts 1 and 2 based on average NCE score on the Terra Nova. 
Note: Figures for Cohorts 3 and 4 based on average scale scores on the MSA. 



Outcomes for KIPP Students 

Attendance 

KIPP students had higher average fifth grade attendance rates than comparison students 
(see Table 3). As comparison students transitioned primarily to middle schools in grade 6,"^ their 
attendance rates declined and the gap between KIPP and comparison students grew wider.^ 

KIPP student attendance also declined slightly as students moved into the middle grades (6-8), 
but not as dramatically as for comparison students. KIPP student attendance did not dip below 
93%, while the attendance rate for comparison school students in the middle grades ranged from 
85-89%. 



Table 3. Summary of Attendance Outcomes for KIPP and Comparison Students 





Cohort 1 
5“* grade 2002-03 


Cohort 2 
5“* grade 2003-04 


Cohort 3 
5‘*’ grade 2004-05 


Cohort 4 
5“* grade 2005-06 




Original 

KIPP 

Cohort 


Comparison 

Group 


Original 

KIPP 

Cohort 


Comparison 

Group 


Original 

KIPP Comparison 
Cohort Group 


Original 

KIPP Comparison 
Cohort Group 


Year I (5“) 


99.6 


94.0 


94.7 


93.8 


97.0 94.3 


96.2 95.2 


Year 2 (on time 6*) 


96.4 


86.8 


96.9 


85.1 


96.9 89.7 




Year 3 (on time 7*) 


96.3 


85.5 


93.8 


85.7 






Year 4 (on time 8*) 


96.6 


85.7 











Note: Statistically significant differences (p <.05) indicated in bold font. 



See Appendix B for a chart summarizing the middle grade schools attended by comparison group students in 6* 
grade. 

^ Transportation by the district (yellow school bus) was provided for KIPP students but not for regular education 
comparison students. 








Achievement Scores 



Tables 4 to 7 present average scores on the Maryland School Assessment (MSA) for 
KIPP and comparison school students in each cohort who were promoted on schedule. We 
distinguish between two KIPP student categories: KIPP students who stayed all year at each 
grade level, and students who left during a particular year and grade level.^ We also present 
average results for the group of students who transferred into KIPP each particular year (5* and 
6* grades). 



® N’s for students leaving KIPP in Tables 4 to 7 do not necessarily match N’s in Figures 3 to 6 (depicting attrition) 
because they represent those students on grade level for whom test scores were available. 
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Table 4.1 Cohort 1 MSA Math Scores by KIPP Groups and Comparison Group 

Original KIPP Cohort 

Stayed at Left KIPP Transferred Regularly promoted 

KIPP all year N that year N into KIPP N comparison group 
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S06 8th grade Avg. Score 403.9 40 395.5 5 NA 0 377.7 303 

% proficient or above 77 . 5 % 40 60 . 0 % 5 NA 0 35 . 0 % 303 




Table 5.1 Cohort 2 MSA Math Scores by KIPP Groups and Comparison Group 

Original KIPP Cohort 

Stayed at KIPP Left KIPP that Transferred Regularly promoted 
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S06 7th grade Avg. Score 411.7 43 395.9 9 NA 0 369.6 307 

% proficient or above 81.4% 43 55.6% 9 NA 0 32.9% 307 




Table 6.1 Cohort 3 MSA Math Scores by KIPP Groups and Comparison Group 

Original KIPP Cohort 

Stayed at KIPP Left KIPP that Transferred Regularly promoted 
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Table 7.1 Cohort 4 MSA Math Scores by KIPP Groups and Comparison Group 

Original KIPP Cohort 
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Because of some pre-existing differences between KIPP and comparison students in 
some cohorts, we conducted multivariate analyses for each cohort/grade level that 
controlled for 4^** grade test score, gender, and special education status. Analyses used 
two different versions of the KIPP treatment variable: a stringent “intent to treat” 
variable that coded all students who were enrolled at KIPP as of September 30 of the 5* 
grade year as KIPP students (even though they may have transferred out of KIPP soon 
afterwards); and a more nuanced (and less stringent) variable that measured whether or 
not the student finished that particular school year at KIPP. Table 8 summarizes the 
results of these analyses, indicating whether or not a significant effect of KIPP treatment 
was found for each cohort and grade level. (See Appendix C for full regression analysis 
results.) Even taking into account some pre-existing differences between KIPP and 
comparison students, analyses showed that KIPP students generally outperformed 
comparison school students on achievement measures. In particular: 

• KIPP students significantly outperformed their feeder school comparison group in 
5th grade mathematics every year (even controlling for higher prior achievement 
in math). But KIPP students did not outperform their feeder school comparison 
group in 5* grade reading achievement. 

• In grades 6 to 8, when most of the comparison group students attended large 
middle schools, KIPP students outperformed comparison students in reading and 
in math. In most cases, the KIPP effect was significantly positive even when 
students who had transferred out of KIPP and were still at the on-time grade level 
were included in the analyses as “ever KIPP” students. The necessary exclusion 
of retained students from test score analyses may have an unmeasured impact on 
these reported results. 



Table 8. KIPP and Comparison Group Cohort Achievement Summary^ 





Cohort 1 


Cohort 2 


Cohort 3 


Cohort 4 




S"* grade 2002-03 


S"* grade 2003-04 


s'” grade 2004-05 


s'” grade 2005-06 




Read 


Math 


Read 


Math 


Read 


Math 


Read 


Math 


4* grade prior 


No Difference 


No 

Difference 


KIPP 

Higher 


KIPP 

Higher 


No 

Difference 


KIPP 

Higher 


No 

Difference 


No 

Difference 


5* grade 


No Difference 


KIPP 

Higher 


No 

Difference 


KIPP 

Higher 


No 

Difference 


KIPP 

Higher 


KIPP 

Lower** 


KIPP 

Higher 


6* grade 


KIPP Higher 


KIPP 


KIPP 


KIPP 


KIPP 


KIPP 






(excl. drops) 


Higher 


Higher 


Higher 


Higher 


Higher 






7* grade 


KIPP Higher 


KIPP 

Higher 


KIPP 

Higher 


KIPP 

Higher 










8* grade 


KIPP Higher 


KIPP 

Higher 















^ We used ordinary least squares regression in these analyses. Even though students were clustered in 
schools, particularly in 5* grade, the number of 5* grade schools was too small for hierarchical linear 
models, and in grades 6 to 8 the comparison students were scattered sparsely (n=l or n=2) in many 
different schools (though there were a couple of concentrations at two large middle schools). 
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Achievement Growth in Cohort 1 



The available achievement measures allow analyses of achievement growth only for 
the first KIPP cohort from spring of the prior 4* grade year to spring of the 5* grade year 
(2002 and 2003). These analyses analyze growth on the CTBS/5 Terra Nova between 4* 
and 5* grades for KIPP and comparison group students. Table 9 summarizes normal 
curve equivalent (NCE) gains for students with scores in both years for both reading and 
math. 

On average, KIPP students gained 24 NCE points on the Terra Nova math composite 
score between spring of 4* grade and spring of 5* grade (one year in KIPP school), 
compared to 0.7 NCE points for the comparison group (most of whom remained in the 
KIPP feeder elementary schools). KIPP students gained slightly more in computation 
(24.8 NCEs) than in math concepts (17.1 NCEs). By contrast, there was no significant 
difference between KIPP students and comparison students on reading growth between 
4* and 5* grade. On average, KIPP students lost 0.8 NCEs, while comparison students 
gained 1.0 NCEs during that year. When reading scores are decomposed KIPP students 
gained 4.3 NCEs in reading comprehension and lost 4.0 NCEs in vocabulary, while 
comparison students gained about 1 NCE on each subtest. 

Table 9. Summary of 4* to 5* Grade NCE Achievement Growth for KIPP and 



Comparison Students 





Reading 

Composite 


Vocabulary 


Reading 

Comprehension 


Math 

Composite 


Computation 


Math 

Concepts 


KIPP students 


0.-8 


-4.0 


4.3 


24.0 


24.8 


17.1 


Comparison group 


1.0 


1.2 


1.2 


0.7 


1.2 


-0.1 



Attrition 

Analyses indicate non-trivial levels of attrition among the original KIPP cohorts, 
occurring not only during the 5* grade year but in subsequent years as well. Eigures 3 to 
6 summarize the patterns of attrition (and partial replacement) for each of the four 
cohorts. Eigure 7 depicts attrition and retention outcomes overall. There was a slight 
tendency for attrition to be higher among males than females (Eigure 8). 



^ Note: 4th grade prior achievement results based on analysis of variance results; 5* and higher grade effects 
based on regression analyses of test scores (including only students in the on-time grade level), controlling 
for 4* grade prior achievement, gender, and special education status. Except where noted, the “KIPP 
higher” results are based on the most stringent test (including all students ever in KIPP with those who had 
transferred out of KIPP). The “no difference” or “KIPP lower” results included only those KIPP students 
who remained until the end of the year. 

** Students leaving KIPP during 05-06 scored significantly lower than comparison group on MSA reading that 
year. KIPP stayers had a lower score than comparison group, but not quite significant (p=.07). Analyzed 
together, KIPP students scored significantly worse than the comparison group 
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Figure 3. Cohort 1: 2002-03 through 2005-06 



Transferred in 
during 2002-03 

5 (all promoted 4*) 




Transferred in 
during 2003-04 

2 (first time 6*) 




2 leavers: 1 02-03 
transferred as 5th; 
1 03-04 transfer 
as 6th 





▼ 
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Figure 4. Cohort 2; 2003-04 through 2005-06^ 




Left during 
2003-04 

16 (alls*) 



Left during 
2004-05 



10 (3 5*, 7 6*) 



3 leavers: 1 03-04 
transfer as 7th, 2 
04-05 transfers as 
7th 




Left during 
2005-06 



19 (5 6*, 14 7*) 



Final at End of 
2005-06 

61-6 6*, 55 7th 

(44 original members, 
17 transfers) 



9 

In Figures 3-5, students coded as leaving during a particular year did not have a BCPSS transfer or withdrawal 
code during the previous year, but may have left KIPP prior to the beginning of the school year in which they were 
coded as leaving. 
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Figure 5. Cohort 3: 2004-05 through 2005-06 




Left during 
2004-05 



2 (all 5*) 



Left during 
2005-06 

16 (2 5*, 14 6th) 



Final at End of 
2005-06 
81-5 6‘^75 

8th 

(67 original members, 
14 transfers) 
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Figure 6. Cohort 4: 2005-2006 



Transferred in 
during 2005-06 

12 (2 out of district, 
10 from feeders) 



1 1 leavers as 5th 



2005-06 Total 
105 - all 5“* 

(93 original members, 
1 2 transfers) 








Left during 






2005-06 




r 


19 (all 5*) 



Final at End of 
2005-06 
75 - all 5“* 

(74 original members, 
1 transfer) 
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Figure 7. Summary of Attrition and Retention in Four BCPSS KIPP Cohorts (2002-03 to 2005-06) 
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Figure 8. Percentage of Females 
Over Time in Each KIPP Cohort 



70.0% 

60.0% 

50.0% 

40.0% 

30.0% 

20 . 0 % 

10 . 0 % 

0 . 0 % 

Cohort 1 Cohort 2 Cohort 3 Cohort 4 




□ Original 

□ End Y1 

□ End Y2 

□ End Y3 
■ End Y4 



Of the 79 fifth graders enrolled (as of September 30) in KIPP’s first year (2002-03), just 
49 students remained at KIPP four years later (Spring 2006). Of these, just 40 were 8* graders; 
the other 9 had been retained and were in grade. A total of 7 students had transferred into this 
cohort (5 in Year 1, 2 in Year 2), with 2 leaving before the end of 2005-06. 

In KIPP’s second year (2003-04), a total of 89 new fifth graders were on roll as of 
September 30. By the end of their third year, just 44 of those students remained (all in 7* grade 
but one). A total of 28 students transferred into this cohort (3 in Year 1, 25 in Year 2), with 1 1 
transfer students leaving before the end of 2005-06. 

A total of 85 new 5^*^ grade students enrolled in KIPP in 2004-05 (KIPP’s third year). All 
but two of these came back the next year (with 7 repeating 5* grade and one skipping 6* grade to 
enroll in 7*), but 13 students left during that second year. At the end of 2005-06, just 67 
students were still enrolled (mostly in 6^*^ grade, with 5 fifth graders and one seventh grader). All 
six transfer students into this 5* grade cohort in 2004-05 left by the end of that year, but 14 of 17 
students who transferred into the cohort in 2005-06 remained at the end of that year. 

KIPP’s fifth grade cohort in 2005-06 began with 93 students as of September 30, but 19 
of these had transferred by the end of the year (with 74 students remaining). There was an 
additional transfer student (of 12 total transfer students during the year) who remained at the end 
of the year. 

Students who left KIPP did not have significantly different prior achievement (4* grade) 
scores, but did have significantly lower 5^’’ grade scores as a group. Since achievement of 
students who left during the 5* grade year could be due to instruction received in the post- KIPP 
school they attended, we excluded first-year leavers from analyses. The group of students who 
left KIPP after their first year (and had all of 5* grade at KIPP) still scored significantly lower on 
some 5* grade achievement measures than did students who stayed. Table 10 summarizes these 
differences for each cohort. In Cohort 1 , leavers scored significantly lower than stayers on 5* 
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grade reading. In Cohort 2, leavers scored significantly lower than stayers on 5* grade math. In 
Cohort 3, leavers had significantly lower 5^** grade attendance at KIPP than did stayers, and they 
also had significantly lower 5* grade reading scores. 

Table 10. KIPP Cohort Achievement Summary for Stayers and Leavers 
(Excluding Those who Left During 5*** Grade Year) 



Cohort 1 Cohort 2 Cohort 3 

5*** grade 2002-03 5**^ grade 2003-04 5^“ grade 2004-05 



5* grade Math 


Stayers 


Eeavers 


Stayers 


Eeavers 


Stayers 


Eeavers 


405.1 


397.8 


427.0 


414.0 


435.4 


424.4 


5* Grade Reading 


414.4 


397.3 


387.0 


383.3 


402.5 


379.2 


5* grade attendance 


99.8% 


99.4% 


95.7% 


93.3% 


97.7% 


93.7% 



Note: Statistically significant differences (p <.05) indicated in bold font. 



Those students who withdrew from KIPP left the school for a variety of destinations. 
Many of those who withdrew during their first year at KIPP returned to a feeder elementary 
school, as is evidenced in Table 11. The large majority of all students who left KIPP transferred 
to Baltimore public school. It is possible that some of these students subsequently opted to leave 
the district, but our analyses did not follow their trajectories over time. The majority of the 
remaining students withdrawing from KIPP left BCPSS for a Maryland public school or, to a 
lesser extent, a public school outside of Maryland. Few students opted for home schooling (only 
1 student) or for private schools in Baltimore (2 students). Table 11 elaborates on students’ 
destinations by cohort. 

Since most comparison group students had a necessary change of school (into middle 
school) during the longitudinal study, it was not possible to calculate a directly comparable rate 
of attrition for this group. We can compare withdrawal rates reported by the Maryland State 
Department of Education (MSDE) for middle grades (6-8) students at KIPP and BCPSS as a 
whole. The elementary withdrawal rate for KIPP reported by MSDE applies to grade 5 only, but 
there is no reported 5* grade withdrawal rate for BCPSS. We estimated a fifth grade withdrawal 
rate for BCPSS for each year by finding the percentage of 5* grade students in each cohort year 
who had multiple records in the BCPSS database (almost always a sign of change of school) or a 
withdrawal code (leaving district) for those with just one record. These figures are summarized 
in Table 12 below. While the MSDE reported middle grades withdrawal rate for KIPP is 
considerably lower than for BCPSS as a whole, the 5* grade withdrawal rate for KIPP is 
generally higher (particularly in 2005-06) than for BCPSS as a whole. The small size of KIPP 
and the fact that there are no entrants to KIPP at the 7* and 8* grade level make it somewhat 
problematic to compare it to (primarily) middle school withdrawal rates. KIPP’s withdrawal 
rates for grades 6-8 are similar to those of Baltimore charter schools serving grades 6-8 (though 
as documented in Appendix D, KIPP’s 2005-06 withdrawal rate tends to be higher than most 
other charter-type schools). 



There was no evidence that comparison group students leaving the district had higher prior achievement 
scores than those remaining in the district (a factor which could have contributed to the KIPP achievement effect, 
but apparently did not). 
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Table 11. Initial Withdrawal Destinations for Students Leaving KIPP 





Cohort 1 


Cohort 2 


Cohort 3 


Cohort 4 


To a feeder school 


33.33% 


10 


26.67% 


12 


16.67% 


3 


68.42% 


13 


To another BCPSS school 


56.67% 


17 


48.89% 


22 


55.56% 


10 


21.05% 


4 


Elementary Sehools 


















Commodore John Rodgers Elementary (27) 
Walter P. Carter Elementary (134) 








2 








2 


Lafayette Elementary (202) 
















1 


Belmont Elementary (217) 
Callaway Elementary (251)* 




1 




1 








1 


Elementary/Middle Sehools 


















Cherry Hill Elementary /Middle (159) 












1 






Dickey Hill Elementary Middle (201) 
Roland Park Elementary Middle (233) 
Glenmount Elementary Middle (235) 




1 




1 




1 






Middle Schools 


















Hamilton Middle (41) 
Garrison Middle (42) 




2 




2 










Highlandtown Middle (43) 
Robert Poole Middle (54) 
William H. Lemmel Middle (79) 




1 




1 




3 






Dr. Roland Patterson Sr. Academy (82) 




11 




8 




3 






Paul Laurence Dunbar Middle (133) 








1 










Winston Middle (209) 








1 










Pimlico Middle (222) 




1 




4 




2 






To Baltimore private school 


3.33% 


1 


0.00% 


0 


5.56% 


1 


0.00% 


0 


To Maryland public school 


6.67% 


2 


13.33% 


6 


16.67% 


3 


5.26% 


1 


To US public school 


0.00% 


0 


6.67% 


3 


5.56% 


1 


5.26% 


1 


To home schooling 


0.00% 


0 


2.22% 


1 


0.00% 


0 


0.00% 


0 


Unknown 


0.00% 


0 


2.22% 


1 


0.00% 


0 


0.00% 


0 


Total 


100.0% 


30 


100.0% 


45 


100.0% 


18 


100.0% 


19 


*This school later became a feeder school but was not for the first two cohorts 
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Table 12. Yearly Withdrawal (Including Within District Transfer) Rates for KIPP and BCPSS 
Students “ 





2002-03 


2003-04 


2004-05 


2005-06 




gth 


h-S"* 


5*” 


6-8“* 


^th 




^th 


6-8th 


KIPP 


18.0% 


NR 


17.8% 


NR 


14.2% 


9.4% 


33.1% 


12.6% 


BCPSS 


15.7% 


24.5% 


16.2% 


27.5% 


17.4% 


23.0% 


16.7% 


20.8% 



On-Time Promotion 

Table 13 summarizes the on-time promotion rates for KIPP and comparison group 
students for each grade in each cohort. In Cohort 1, on-time promotion rates to 6* and 7* grades 
were somewhat (but not significantly) higher for KIPP. KIPP had a slightly (but not 
significantly) lower on-time promotion rate to 8* grade in Cohort 1. In Cohorts 2 and 3, KIPP 
students had a significantly lower on-time promotion rate to 6* grade than did comparison group 
students. KIPP’s lower on-time promotion rate to 7^*' grade in Cohort 2 approached statistical 
significance (p=.066). KIPP’s standards for promotion were higher than those in comparison 
students’ schools (with 70% rather than 60% as the lowest passing grade). 



Table 13. Summary of On-Time Promotion Rates 





Cohort I 
5* grade 2002-03 


Cohort 2 
5“* grade 2003-04 


Cohort 3 
5‘** grade 2004-05 




Original 


Comparison 


Original 


Comparison 


Original 


Comparison 




Cohort 


Group 


Cohort 


Group 


Cohort 


Group 


On-time 


97.4 


93.6 


93.1 


98.6 


91.7 


98.6 


promotion to 6th 
On-time 

promotion to 7th 
On-time 

promotion to 8th 


89.5 

79.0 


87.2 

84.0 


83.5 


90.4 



Note; Statistically significant differences (p<.l) indicated in bold font. 



‘ ‘ The figures in Table 12 for KIPP 5* , KIPP 6-8, and BCPSS 6-8 are from the MSDE website. 
Withdrawals are defined there as “the number and percentage of students withdrawing (transfers and terminations) 
for any reason during the September to June school year after the first day of school. The percentage of withdrawals 
is calculated by dividing the number of withdrawals by the average daily membership.” The BCPSS 5* grade 
withdrawal figure in the table is estimated from individual level BCPSS student data (% of 5* grade students with 
multiple records or a record of transfer out of the district that year). 
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Conclusion 



Longitudinal analyses of four cohorts of students in Baltimore indicate significantly 
better achievement outcomes for KIPP students, particularly in mathematics, than for 
comparison group students from the same feeder elementary schools. Even when students who 
left KIPP are included as KIPP students in analyses, achievement results are significantly better 
for the KIPP group. For those students who remained at KIPP during the middle grades (6* 
through outcomes were dramatically better than for those who attended other district middle 
grades schools. 

Though KIPP and comparison students were similar in 4* grade achievement and 
attendance, as well as on demographic variables, it is important to note that it was not possible to 
measure other important differences between KIPP and comparison students. In particular, 
differences in family support variables (parental education, amount of parental interaction with 
students on academic and other activities, etc.) could not be measured, and could contribute 
significantly to the later achievement differences between the groups. 

It is likely that several components of the KIPP program contributed to higher student 
achievement: the longer school day and other additional hours of instruction, high quality 
curriculum and instruction (especially in mathematics), and positive school climate (facilitated 
by smaller numbers of students than in comparison students’ schools, fewer behavioral problems, 
etc.). Scaling up these components throughout the district is an attractive proposal, but would 
likely be very costly. It is also important not to ignore the attrition from the KIPP program, as 
well as the relatively higher retention in grade rates for KIPP students. Assuring that a KIPP 
model could succeed on a large scale and sustain such high rates of achievement would be an 
extremely challenging task. Qualitative studies that probe into reasons for the attrition among 
KIPP students would be a useful research contribution as district policymakers make decisions 
on expanding the number of KIPP schools. 

It is possible that keeping middle grades students in smaller learning environments 
(newly converted K-8 schools rather than middle schools) will address the school climate issue, 
but the primary challenge is to find ways to ensure high quality instruction in every classroom, 
every day. In addition, more ways of providing extra help for struggling students need to be 
incorporated into district- and school-level planning of instructional delivery. The KIPP Ujima 
Village Academy in Baltimore has shown that high quality instruction and extra learning time in 
a positive school environment does make a difference in student achievement. The challenge is 
to find ways to make this a reality for most urban students, rather than for just a few. 
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Appendix A 



The following chart summarizes elementary schools in the designated geographic zone for KIPP 
Ujima Academy that were the sending 4* grade schools for KIPP students in each 5* grade 
cohort. 



KIPP Feeder School Cohort Summary 







% 


Cohort 1 


Cohort 2 


Cohort 3 


Cohort 4 


School 


School 


FRL 


5‘** grade 


5‘** grade 


5* grade 


5* grade 




Number 


04-05 


2002-03 


2003-04 


2004-05 


2005-06 


Brehms Lane 


231 


81.1% 


X 








Eutaw- 

Marshbum 


11 


91.5% 


X 








Edgecombe 

Circle 


62 


87.7% 


X 


X 


X 


X 


Langston 

Hughes 


5 


96.7% 


X 


X 


X 


X 


MLK Jr. 


254 


88.5% 


X 


X 


X 


X 


Pimlico 


223 


91.0% 


X 


X 


X 


X 


Arlington 


234 


83.1% 






X 


X 


Callaway 


251 


81.4% 






X 


X 


Cross Country 


247 


68.4% 






X 


X 



The comparison group for each cohort (except Cohort 2) included promoted 4* graders from the 
marked schools. For cohort 2, we included students from schools #231 and #1 1 to maintain n 
size. Prior achievement scores from those schools did not differ significantly from those from 
the others. In cohorts 3 and 4 excluded #231 and #11 from comparison group, and included 
students from the new feeder elementaries. 
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Appendix B 



Comparison Students' 6th Grade School of Attendance, By Cohort 






Cohort 1 


Cohort 2 


Cohort 3 




Elementary & Elementary/Middle 








4 


Steuart Hill Elementary 




1 


2 


8 


City Springs Elementary 






1 


31 


Coldstream Park Elementary 






1 


51 


Waverly Elementary 


1 


2 


2 


81 


North Bend Elementary 


1 


1 




89 


Rognel Heights Elementary 




1 


3 


97 


Collington Square Elementary 


1 






105 


Moravia Park Primary 


1 


1 


2 


157 


George G. Kelson Elementary 


1 




4 


247 


Cross Country Elementary 






47 


12 


Lakeland Elementary/Middle 


1 


2 


2 


54 


Barclay Elementary/Middle 




5 




58 


Dr. Nathan A. Pitts Asburton Elementary/Middle 


2 




4 


66 


Mount Royal Elementary/Middle 


5 


6 


6 


76 


Francis Scott Key Elementary/Middle 


2 




1 


163 


Patapsco Elementary/Middle 




1 




164 


Arundel Elementary/Middle 


1 






205 


Dickey Hill Elementary/Middle 


1 


1 




210 


Hazelwood Elementary/Middle 


1 


6 


2 


214 


Guilford Elementary/Middle 


4 




2 


233 


Roland Park Elementary/Middle 


6 


7 


20 


235 


Glenmount Elementary/Middle 


3 


4 


1 


236 


Hamilton Elementary/Middle 


3 




1 




Middle 








41 


Hamilton Middle 


13 


19 


2 


42 


Garrison Middle 


9 


17 


40 


43 


Highlandtown Middle 


12 


4 


3 


46 


Chinquapin Middle 


5 


8 


6 


49 


Northeast Middle 


6 


10 


2 


56 


Robert Poole Middle 


3 


4 


4 


57 


Lombard Middle 


2 


5 


5 


75 


Calverton Middle 


4 


7 


6 


78 


Harlem Park Middle 


4 


3 


6 


79 


William H. Lemmel Middle 


7 


6 


11 


80 


West Baltimore Middle 


4 


3 


11 


82 


Dr. Roland N. Patterson Sr. Academy 


130 


143 


119 


130 


Booker T. Washington Middle 


24 


32 


31 


133 


Paul Laurence Dunbar Middle 


9 






162 


Diggs-Johnson Middle 


2 


5 


2 


170 


Thurgood Marshall Middle 


77 


65 


2 


209 


Winston Middle 


1 




4 


222 


Pimlico Middle 


42 


29 


82 


230 


Canton Middle 




4 


1 


239 


Benjamin Franklin Jr. Middle 


2 




1 


255 


Southeast Middle 






2 




Other/Unknown 








201 


Francis M. Wood Alternative High 


2 


2 


6 


303 


Upton School 


1 






321 


Midtown Academy 


1 






325 


Connexions Community Leadership 


1 


1 




488 


Alternative Learning Center 


1 




1 
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Note: 5th grade results are based on NCE scores on Terra Nova tests; 6th-8th grade results based on scale scores on MSA tests. 
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Male -5.61 4.44 .208 -7.56 2.93 .010 

Ever KIPP 67.49 5.94 .000 23.88 3.94 .000 
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Appendix C-4 



Summary of Ordinary Least Squares Regression Resuits for Reading and 
Mathematics Achievement Scores for Cohort 4 





5th Grade Math 
2006 

Coeff (se) Sig. 


5th Grade Reading 
2006 

Coeff (se) Sig. 


Constant 


185.63 


14.19 


.000 


204.55 


13.33 


.000 


4th grade score 


0.57 


0.04 


.000 


0.51 


.03 


.000 


Ever Special Education 


-13.17 


4.13 


.002 


-6.22 


3.10 


.046 


Male 


-4.95 


3.14 


.116 


-3.31 


2.40 


.169 


Ever KiPP 


8.30 


3.95 


.036 


-7.33 


3.01 


.015 


Separating KIPP leavers 














(reported if EVERKIPP does 


not yield significant positive effect) 






Constant 








204.84 


13.32 


.000 


4th grade score 








0.51 


.03 


.000 


Ever Special Education 








-5.96 


3.10 


.056 


Male 








-3.16 


2.40 


.188 


KiPP stayer 








-5.80 


3.20 


.071 


KiPP ieaver 








-16.58 


7.31 


.024 



32 






Appendix D 



MSDE-Reported Middle Grades (e*** to 8“*) Withdrawal Rates 
for KIPP Ujima Village Academy and Other Charter and Charter-Type Schools 





2004-05 


2005-06 


KIPP Ujima Village 


9.4% 


12.6% 


Midtown Academy 


12.3% 


6.6% 


Connexions 


4.9% 


2.6% 


City Springs 


14.5% 


9.2% 


Collington Square 


12.8% 


17.2% 


Hampstead Hill 


NA 


9.5% 


Crossroads 


10.2% 


6.3% 


New Song Academy 


4.9% 


0.0% 


Stadium School 


8.0% 


9.7% 
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